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Abstract — Most medical images are now digitized and stored 
with semantic information, leading to medical case databases. 
They may be used for aid to diagnosis, by retrieving similar 
cases to those in examination. But the information are often 
incomplete, uncertain and sometimes conflicting, so difficult 
to use. In this paper, we present a Case Based Reasoning 
(CBR) system for medical case retrieval, derived from the 
Dezert-Smarandache theory, which is well suited to handle 
those problems. We introduce a case retrieval specific frame 
of discernment 6 , which associates each element of 0 with 
a case in the database; we take advantage of the flexibility 
offered by the DSmT’s hybrid models to finely model the 
database. The system is designed so that heterogeneous sources 
of information can be integrated in the system: in particular 
images, indexed by their digital content, and symbolic infor- 
mation. The method is evaluated on two classified databases: 
one for diabetic retinopathy follow-up (DRD) and one for 
screening mammography (DDSM). On these databases, results 
are promising: the retrieval precision at five reaches 81.8% on 
DRD and 84.8% on DDSM. 

Index Terms — Case based reasoning, Image indexing, Dezert- 
Smarandache theory, Diabetic Retinopathy, Mammography 

I. INTRODUCTION 

In medicine, the knowledge of experts is a mixture of 
textbook knowledge and experience through real life clinical 
cases. Consequently, there is a growing interest in case- 
based reasoning (CBR), introduced in the early 1980s, for 
the development of medical decision support systems [1]. 
The underlying idea of CBR is the assumption that analogous 
problems have similar solutions, an idea backed up by physi- 
cians’ experience. In CBR, the basic process of interpreting 
a new situation revolves around the retrieval of relevant cases 
in a case database. The retrieved cases are then used to help 
interpreting the new one. 

We propose in this article a CBR system for the retrieval 
of medical cases made up of a series of images with 
contextual information: a class of CBR problems which has 
hardly been treated. The proposed system is applied to a 
Diabetic Retinopathy (DR) multimedia database built up in 
our laboratory; to diagnose DR, physicians analyze series 
of multimodal photographs together with contextual infor- 
mation such as the patient age, sex and medical history. To 
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show that the method is generic, we also applied it to DDSM, 
a public access database for screening mammography; to 
screen mammography, physicians analyze two views of each 
breast, with associated contextual information. 

When designing a CBR system to retrieve such cases, several 
problems arise. We have to aggregate heterogeneous sources 
of evidence (images, contextual information) and to man- 
age missing information. These sources may be uncertain 
and conflicting. As a consequence, we applied the Dezert- 
Smarandache Theory (DSmT) of plausible and paradoxical 
reasoning, proposed in recent years [2], which is well suited 
to fuse uncertain, highly conflicting and imprecise sources 
of evidence. 

II. MATERIAL AND METHODS 

A. Databases 

1 ) Diabetic retinopathy database: the diabetic retinopathy 
(DR) database contains retinal images of diabetic patients, 
with associated anonymous information on the pathology. 
Diabetes is a metabolic disorder characterized by sustained 
inappropriate high blood sugar levels. This progressively 
affects blood vessels in many organs, which may lead to 
serious renal, cardiovascular, cerebral and also retinal com- 
plications. Different lesions appear on the damaged vessels, 
which may lead to blindness. The database is made up of 63 
patient files containing 1045 photographs altogether. Images 
have a definition of 1280 pixels/line for 1008 lines/image. 
They are lossless compressed images. Patients have been 
recruited at Brest University Hospital since June 2003 and 
images were acquired by experts using a Topcon Retinal 
Digital Camera (TRC-50IA) connected to a computer. An 
example of an image series is given in figure 1. 




Fig. 1. Photograph series of a patient eye 
Images (a), (b) and (c) are photographs obtained by applying 
different color filters. Images (d) to (j) form a temporal angiographic 
series: a contrast product is injected and photographs are taken at 
different stages (early (d), intermediate (e)-(i) and late (j)). 
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The contextual information available is the patients’ age 
and sex and structured medical information (about the gen- 
eral clinical context, the diabetes context, eye symptoms 
and maculopathy). Thus, at most, patients records are made 
up of 10 images per eye (see figure 1) and of 13 contex- 
tual attributes; 12.1% of these images and 40.5% of these 
contextual attribute values are missing. The disease severity 
level, according to ICDRS classification [3], was determined 
by experts for each patient. The distribution of the disease 
severity among the above-mentioned 63 patients is given in 
table I. 

2) Digital Database for Screening Mammography 
(DDSM): the DDSM project [4] has built a mammographic 
image database for research on breast cancer screening. It is 
made up of 2277 patient files. Each one includes two images 
of each breast, associated with some patient information 
(age at time of study, rating for abnormalities, American 
College of Radiology breast density rating and keyword 
description of abnormalities) and imaging information. The 
following contextual attributes are taken into account in the 
system: 

• the age at time of study 

• breast density rating 

Images have a varying definition, of about 2000 pixels/line 
for 5000 lines/image. An example of image sequence is given 
in figure 2. Each patient file has been graded by a physician. 
Patients are then classified in three groups: normal, benign 
and cancer. The distribution of grades among the patients is 
given in table I. 




Fig. 2. Mammographic image sequence of the same patient, (a) and (b) 
are two views of the left breast, (c) and (d) are two views of the right one. 



B. Including images in the retrieval system 

To include images in the proposed retrieval system, we 
have to define a distance measure between images and to 
cluster images acquired at a given imaging modality into 
a finite number of groups. For this purpose, we follow the 
usual steps of Content-Based Image Retrieval (CBIR) [5]: 1) 
building a signature for each image (i.e. extracting a feature 
vector summarizing their numerical content), and 2) defining 
a distance measure between two signatures. Thus, measuring 
the distance between two images comes down to measuring 
the distance between two signatures. We can then cluster 



TABLE I 

Patient disease severity distribution 



database 


disease severity 


number of 
patients 




no apparent diabetic retinopathy 


7 




mild non-proliferative 


9 


DRD 


moderate non-proliferative 


22 




severe non-proliferative 


9 




proliferative 


9 




treated/non active diabetic retinopathy 


11 




normal 


695 


DDSM 


benign 


669 




cancer 


913 



similar image signatures according to the defined distance 
measure. 

In previous studies, we proposed to compute a signature 
for images from their wavelet transform (WT) [6]. These 
signatures model the distribution of the WT coefficients in 
each subband of the decomposition. The associated distance 
measure d [6] computes the divergence between these dis- 
tributions. We used these signature and distance measure to 
cluster similar images. 

Any clustering algorithm can be used, provided that the dis- 
tance measure between feature vectors can be specified. We 
used FCM ( Fuzzy C-Means) [7], one of the most common 
algorithms, and replaced the Euclidian distance by d. 

C. Dezert-Smarandache Theory 

The Dezert-Smarandache Theory (DSmT) allows to com- 
bine any type of independent sources of information repre- 
sented in term of belief functions. It is more general than 
probabilistic fusion or Dempster- Shafer theory (DST). It is 
particularly well suited to fuse uncertain, highly conflicting 
and imprecise sources of evidence [2]. 

1) The fusion model: let 0 = {0i,02 ? ---} be a set of 
hypotheses under consideration for a fusion problem; 0 is 
called the frame of discernment. In DST, these hypothe- 
ses are assumed incompatibles (constrained model M°(0)), 
while in DSmT they are not (free model (0)): if 0 = { 
“blue” , “red” }, we may model objects that are both blue 
and red (magenta). Thus, in DSmT, a belief mass m(A) is 
assigned to each element A of the hyper-power set D(6), 
i.e. the set of all composite propositions built from elements 
of 6 with n and U operators, such that m(0) = 0 and 
Xmg£>(< 9) m (^) = 1; m is called a generalized basic belief 
assignment (gbba). Nevertheless, it is possible to introduce 
constraints in the model [2] (hybrid model M(6))\ we can 
specify pairs of incompatible hypotheses (0 aj 0&), be. each 
subset A of 6 a D Ob must have a null mass, noted A e C{6). 

2) The fusion operators and the decision functions: to 
fuse information in the DSmT framework, the user specifies 
a gbba rrij for each source of evidence Sj, j = 1..N. Then, 
these gbba are fused into a global gbba m/, according to a 
given rule of combination. Several rules have been proposed 
to combine mass functions, including the hybrid rule of com- 
bination or the PCR (Proportional Conflict Redistribution) 
rules [2]. 
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Fig. 3. Considering the frame of discernment 9 = { 6 >i, 6 > 2 , $ 3 }, the 
figure represents, from left to right, the Venn diagram of the constrained, 
the free and a hybrid model (in which 63 is incompatible with the other 
two hypotheses). A non-empty intersection between circles means the 
corresponding hypotheses are compatible. 



3) The decision functions: once the fused gbba rrif has 
been computed, a decision function (such as the credibility, 
the plausibility or the pignistic probability) is used to evalu- 
ate the probability of each hypothesis. The pignistic proba- 
bility BetP , a compromise between the other two functions, 
is used since it provides the best system performance: 

BetP(A)= £ (1) 

B i eD(9),B i ^V(9) ^ 

where is the cardinality of Bi in the Venn diagram 

of M(0)\ in the examples of figure 3, Cm°(#i) = 1» 
C Mf (0 1 )=4^ndC M (0 1 ) = 2. 

D. Dezert-Smarandache Theory Based Retrieval 

1 ) Outline of the method: let c q be a case placed as a 
query by the user. We want to rank the cases in the database 
by decreasing order of relevance for c q . In that purpose, for 
each case c* in the database, i = 1..M, we estimate the 
degree of relevance of q for c q , noted DR(ci,c q ); then we 
rank the cases q by decreasing order of DR{c^c q ), and 
the top five results are returned to the user. To compute 
DR(ci,c q ) 9 we see each case descriptor as a source of 
evidence: the degrees of relevance are first estimated for each 
case feature Fj , j = 1..N, noted DRj (q, c q ) (as described in 
section II-D.3), they are then translated into gbba (see section 
II-D.4) which are fused in the DSmT framework. Finally, 
DR(ci , c q ) is estimated from the fused gbba by the pignistic 
probability (see equation 1). The procedure is summed up in 
figure 5. 

2 ) The model: the following frame of discernment is used 

for the fusion problem: 6 = {Ci, C 2 , where C* is 

the hypothesis “c* is relevant for c q , i = 1..M. The cardinal 
of D(6) is hyper-exponential in M (it is majored by 2 2 ). 
As a consequence, from computational considerations, it is 
necessary to include constraints in the model. These con- 
straints are also justified by logical considerations. Indeed, 
it is not possible for the case c q to be similar to both cases 
c a and Cfr if 

• c a and c\) are dissimilar, or 

• c a and c h have different severity levels 

In those cases, we set A £ C(6) \/A c C a D C&. To 
build the model M(6), we first define a non-oriented graph 
G c = (V,E), that we call compatibility graph; each vertex 
v £ V represents an hypothesis and each edge e £ E a 
couple of compatible hypothesis. To build G c , we link the 



hypothesis associated with each case c* to the hypothesis 
associated with its l nearest neighbors at the same severity 
level. The distance measure used to find the nearest neighbors 
is simply a linear combination of heterogeneous distance 
functions (one for each case feature Fj - see section II-B 
for images), managing missing values [8]. We noticed that 
the complexity of the fusion operators mainly depends on the 
cardinality of the largest clique in G c (a clique is a set of 
vertices V such that for every two vertices in V, there exists 
an edge connecting the two). The number l is closely related 
to the cardinality of the largest clique in G c and consequently 
to the complexity of the fusion operation. Finally, the hybrid 
model M{6) is built from G c by identifying all the cliques 
in G c (see figure 4). 





(b) Venn Diagram 
of M(9) 



Fig. 4. Building the model A 4(9) from the compatibility graph. An ex- 
ample of compatibility graph is shown on figure (a). Hypotheses associated 
with cases at different severity levels are represented with different colors. 



3) The degree of relevance: to compute the degree of 
relevance DRj(ci , c q ) that a case q is relevant for c q , given 
the feature Fj, we first define a finite number of states 
fjk for Fj. Then we compare the membership degree of 
Ci (oijkici)) and of c q (otjk{c q )) to each state fjk . If Fj is 
a discrete variable, we associate a state with each possible 
value for Fj and otjk is a Boolean function. If Fj is an image 
attribute, ctjk(c) is the membership degree of a case c to the 
k th image cluster (see section II-B - the same clustering 
algorithm is used for simple continuous variables such as 
age). We assume that the state of the cases in the same class 
are predominantly in a subset of states for Fj. So, in order 
to estimate the conditional probabilities, we use a correlation 
measure Sjk ± k 2 between two feature states / J / Cl and fjk 2 , 
regarding the class of the cases at these states. To compute 
Sjk 1 k 2 > we fi rst compute the mean membership Djk lC (resp. 
Djk 2C ) of cases in a given class c to the state fjk 1 (resp. 
fjk 2 ) (equation 2): 



r> _ a E, ^(a,c)a jfc (a) 
Ujkc ~ P y, x S(x,c) 

£c C =i (iW = i,V(j,fc) 



( 2 ) 



where 5(x, c) = 1 if x is in class c, 5(x,c) = 0 otherwise, 
and (3 is a normalizing factor. Sjk 1 k 2 is given by equation 3: 



S', 



jkik 2 



C 

Djk lC Djk 2 c 

C=1 



(3) 



the expression of DRj(ci,c q ) is given by the following 
equation: 



DRj{ci , c q ) — ^ ^ Oy/ci (Ci)Sjk 1 k 2 (%jk 2 (Cq) (A) 

k i k 2 
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4) The generalized basic belief assignments (gbba): The 
gbba nrij is then defined as follows: 

• We identify the cases {c\)i=\..M'<M such that 
DRj(Ci , C q ) > Tj , 

• We set rrij = m j0 , 

• We set rrij \ C^j = 1 — rrijo. 

To find Tj, we define the following test 7}=“ if DRj(ci , c q ) > 
Tj then Ci is true, otherwise Ci is false”. The sensitivity 
(resp. the specificity) of Tj represents the degree of con- 
fidence in a positive (resp. negative) answer to Tj. Tj is 
relevant if it is both sensitive and specific. As Tj increases, 
sensitivity increases and specificity decreases. So, we set Tj 
as the intersection of the two curves “sensitivity according 
to Tj” and “specificity according to t/\ using a dichotomic 
search. We set rrijo as the sensitivity of Tj. Ui=i Ci is 
assigned the degree of uncertainty of Tj. Note that a single 
setup (Tj, rrijf) is used for each feature, whatever q and c q . 




Fig. 5. Outline of the method 



III. RESULTS 

The mean precision at five (mp5) of the system, i.e. the 
mean number of relevant cases among the top five results, is 
78.6% for DRD and 82.1% for DDSM. As a comparison, the 
mp5 obtained by CBIR (when cases are made up of a single 
image), with the same image signatures, is 46.1% for DRD 
and 70.0% for DDSM [6]. To evaluate the contribution of 
the proposed system for the retrieval of heterogeneous and 
incomplete cases, it is compared to the linear combination of 
heterogeneous distance functions [8] that was used to build 
the model. A mp5 of 52.3% was achieved by this method for 
DRD and of 71.4% for DDSM. To assess the robustness of 
the method regarding missing information, 1) we generated 
artificial cases from each case in the database by removing 
attributes, 2) we placed sequentially each artificial case as 
a query to the system and 3) we plotted on figure 2 the 
precision at five of these queries according to the number of 
available attributes. 

IV. DISCUSSION AND CONCLUSION 

In this article, we introduce a method to include image 
series, with contextual information, in CBR systems. DSmT 
is used to fuse the output of several sensors (cases features). 
We introduced a case retrieval specific frame of discernment 
0, which associates each element of 0 with a case in the 
database; we use the flexibility offered by the DSmT’s hybrid 
models M(0) to finely model the database. On DRD, the 




Fig. 6. Robustness with respect to missing values. Note that cases are 
returned at random when no attributes are inputted (0 on the X axis). 

method largely outperforms our first CBIR algorithm (78.6% 
/ 46.1%). This stands to reason because experts generally 
need more than one image to correctly diagnose the patient’s 
disease severity level. The improvement is also interesting on 
DDSM (82.1% / 70.0%). This non-linear retrieval method is 
significantly more precise than a simple linear combination 
of heterogeneous distances on both databases (78.6% / 52.3% 
on DRD, 82.1% / 71.4% on DDSM). Moreover, if we use a 
Bayesian network to infer the missing values, prior to esti- 
mating the degree of relevance (described in section II-D.3), 
the mp5 becomes 81.8% on DRD and 84.8% on DDSM. It 
is a possible alternative to the decision tree based retrieval 
system we proposed previously [9] (giving a performance of 
79.5% in mp5 on DRD). Linally, the proposed framework is 
also interesting for being generic: any multimedia database 
may be processed, as far as a procedure to cluster cases is 
provided for each new modality (sound, video, etc). 
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